feat: add INLINE + ARROW_STREAM format support for analytics plugin by jamesbroadhead · Pull Request #256 · databricks/appkit

jamesbroadhead · 2026-04-07T15:57:16Z

Summary

Some serverless warehouses only support ARROW_STREAM with INLINE disposition, but the analytics plugin only offered JSON_ARRAY (INLINE) and ARROW_STREAM (EXTERNAL_LINKS)
Adds a new "ARROW_STREAM" format option that uses INLINE disposition, making the plugin compatible with these warehouses
Updates the AnalyticsFormat type to include "ARROW_STREAM"

Test plan

Deploy an app against a warehouse that only supports ARROW_STREAM inline
Verify useAnalyticsQuery with format: "ARROW_STREAM" returns results
Verify existing "JSON" and "ARROW" formats are unaffected

Fixes #242

This pull request was AI-assisted by Isaac.

Some serverless warehouses only support ARROW_STREAM with INLINE disposition, but the analytics plugin only offered JSON_ARRAY (INLINE) and ARROW_STREAM (EXTERNAL_LINKS). This adds a new "ARROW_STREAM" format option that uses INLINE disposition, making the plugin compatible with these warehouses. Fixes #242

Tests verify: - ARROW_STREAM format passes INLINE disposition + ARROW_STREAM format - ARROW format passes EXTERNAL_LINKS disposition + ARROW_STREAM format - Default JSON format does not pass disposition or format overrides

The server-side ARROW_STREAM format added in the previous commit was not exposed to the frontend or typegen: - Add "ARROW_STREAM" to AnalyticsFormat in appkit-ui hooks - Add "arrow_stream" to DataFormat in chart types - Handle "arrow_stream" in useChartData's resolveFormat() - Make typegen resilient to ARROW_STREAM-only warehouses by retrying DESCRIBE QUERY without format when JSON_ARRAY is rejected Co-authored-by: Isaac Signed-off-by: James Broadhead <jamesbroadhead@gmail.com>

…compatibility ARROW_STREAM with INLINE disposition is the only format that works across all warehouse types, including serverless warehouses that reject JSON_ARRAY. Change the default from JSON to ARROW_STREAM throughout: - Server: defaults.ts, analytics plugin request handler - Client: useAnalyticsQuery, UseAnalyticsQueryOptions, useChartData - Tests: update assertions for new default JSON and ARROW formats remain available via explicit format parameter. Co-authored-by: Isaac Signed-off-by: James Broadhead <jamesbroadhead@gmail.com>

When using the default ARROW_STREAM format, the analytics plugin now automatically falls back through formats if the warehouse rejects one: ARROW_STREAM → JSON → ARROW. This handles warehouses that only support a subset of format/disposition combinations without requiring users to know their warehouse's capabilities. Explicit format requests (JSON, ARROW) are respected without fallback. Co-authored-by: Isaac Signed-off-by: James Broadhead <jamesbroadhead@gmail.com>

…ne-support

MarioCadenas · 2026-04-09T09:00:29Z


 /** Supported data formats for analytics queries */
-export type DataFormat = "json" | "arrow" | "auto";
+export type DataFormat = "json" | "arrow" | "arrow_stream" | "auto";


in theory arrow is the same as arrow_stream, so I'm not following what's the problem?

MarioCadenas · 2026-04-09T09:02:51Z

+  /** Format configurations in fallback order. */
+  private static readonly FORMAT_CONFIGS = {
+    ARROW_STREAM: {
+      formatParameters: { disposition: "INLINE", format: "ARROW_STREAM" },


from this URL
https://docs.databricks.com/api/workspace/statementexecution/executestatement#format

Important: The formats ARROW_STREAM and CSV are supported only with EXTERNAL_LINKS disposition. JSON_ARRAY is supported in INLINE and EXTERNAL_LINKS disposition.

so before changing anything this was already supporting arrow, can I know what's the case where this was failing? I would like to see it

MarioCadenas · 2026-04-09T09:15:36Z

Seeing that there's a case of Arrow + inline, let's refactor what we had instead of introducing a new format. Let's change the format "ARROW" to "ARROW_STREAM" and allow it to use both "EXTERNAL_LINKS" and "INLINE". Then for now let's keep JSON + inline as the default. This might require some UI hooks changes too

Previously, _transformDataArray unconditionally called updateWithArrowStatus for any ARROW_STREAM response, which discards inline data and returns only statement_id + status. This was designed for EXTERNAL_LINKS (where data is fetched separately) but broke INLINE disposition where data is in data_array. Changes: - _transformDataArray now checks for data_array before routing to the EXTERNAL_LINKS path: if data_array is present, it falls through to the standard row-to-object transform. - JSON format now explicitly sends JSON_ARRAY + INLINE rather than relying on connector defaults. This prevents the connector default format from leaking into explicit JSON requests. - Connector defaults reverted to JSON_ARRAY for backward compatibility with classic warehouses (the analytics plugin sets formats explicitly). - Added connector-level tests for _transformDataArray covering ARROW_STREAM + INLINE, ARROW_STREAM + EXTERNAL_LINKS, and JSON_ARRAY paths. Co-authored-by: Isaac Signed-off-by: James Broadhead <jamesbroadhead@gmail.com>

Some serverless warehouses return ARROW_STREAM + INLINE results as base64 Arrow IPC in `result.attachment` rather than `result.data_array`. This adds server-side decoding using apache-arrow's tableFromIPC to convert the attachment into row objects, producing the same response shape as JSON_ARRAY regardless of warehouse backend. This abstracts a Databricks internal implementation detail (different warehouses returning different response formats) so app developers get a consistent `type: "result"` response with named row objects. Changes: - Add apache-arrow@21.1.0 as a server dependency (already used client-side) - _transformDataArray detects `attachment` field and decodes via tableFromIPC - Connector tests use real base64 Arrow IPC captured from a live serverless warehouse, covering: classic JSON_ARRAY, classic EXTERNAL_LINKS, serverless INLINE attachment, data_array fallback, and edge cases Co-authored-by: Isaac Signed-off-by: James Broadhead <jamesbroadhead@gmail.com>

jamesbroadhead · 2026-04-16T13:16:35Z

Hi Mario — thanks for the review, you were right on both counts. I've rebased and reworked the PR:

What's happening: Serverless warehouses return format: ARROW_STREAM with disposition: INLINE, putting base64 Arrow IPC data in result.attachment instead of data_array. The docs say ARROW_STREAM only works with EXTERNAL_LINKS, but serverless violates that — and the existing code silently returned empty results for these responses.

What I changed (per your suggestion):

Collapsed the three formats (JSON, ARROW, ARROW_STREAM) into two that match the API enums: JSON_ARRAY and ARROW_STREAM
ARROW_STREAM now supports both INLINE (with attachment decoding) and EXTERNAL_LINKS dispositions
Default remains JSON_ARRAY as you requested
Added format fallback: if ARROW_STREAM+INLINE is rejected by a classic warehouse, it falls back to JSON_ARRAY automatically
Updated the UI hooks to use json_array / arrow_stream naming

Also added 147 new unit tests covering major coverage gaps (service-context 7%→100%, stream-registry 32%→100%, genie connector 61%→97%, files plugin 69%→89%). All 1711 tests pass.

Cleaned up the branch — it's now TS-only, no unrelated changes.

jamesbroadhead added 6 commits April 3, 2026 12:00

Add tests for ARROW_STREAM and ARROW format parameter handling

dd1e1d5

Tests verify: - ARROW_STREAM format passes INLINE disposition + ARROW_STREAM format - ARROW format passes EXTERNAL_LINKS disposition + ARROW_STREAM format - Default JSON format does not pass disposition or format overrides

Merge remote-tracking branch 'origin/main' into fix/arrow-stream-inli…

c3b7318

…ne-support

jamesbroadhead requested a review from MarioCadenas April 7, 2026 16:51

MarioCadenas reviewed Apr 9, 2026

View reviewed changes

jamesbroadhead added 2 commits April 14, 2026 17:15

jamesbroadhead mentioned this pull request Apr 15, 2026

feat: inline Arrow IPC support with format fallback #273

Closed

4 tasks

jamesbroadhead mentioned this pull request Apr 16, 2026

test: close major coverage gaps in service-context, stream-registry, genie connector, files plugin #278

Closed

3 tasks

jamesbroadhead requested a review from MarioCadenas April 16, 2026 16:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add INLINE + ARROW_STREAM format support for analytics plugin#256

feat: add INLINE + ARROW_STREAM format support for analytics plugin#256
jamesbroadhead wants to merge 8 commits intomainfrom
fix/arrow-stream-inline-support

jamesbroadhead commented Apr 7, 2026

Uh oh!

MarioCadenas Apr 9, 2026

Uh oh!

MarioCadenas Apr 9, 2026

Uh oh!

MarioCadenas commented Apr 9, 2026 •

edited

Loading

Uh oh!

jamesbroadhead commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jamesbroadhead commented Apr 7, 2026

Summary

Test plan

Uh oh!

MarioCadenas Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

MarioCadenas Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

MarioCadenas commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jamesbroadhead commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MarioCadenas commented Apr 9, 2026 •

edited

Loading